From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation

In the field of statistical prediction, the tasks of model selection and model evaluation have received extensive treatment in the literature. Among the possible approaches for model selection and evaluation are those based on covariance penalties, which date back to at least 1960s, and are still widely used today. Most of the literature on this topic is based on what we call the “Fixed-X” assumption, where covariate values are assumed to be nonrandom. By contrast, in most modern predictive modeling applications, it is more reasonable to take a “Random-X” view, where the covariate values (both those used in training and for future predictions) are random. In the current work, we study the applicability of covariance penalties in the RandomX setting. We propose a decomposition of Random-X prediction error in which the randomness in the covariates has contributions to both the bias and variance components of the error decomposition. This decomposition is general, and for concreteness, we examine it in detail in the fundamental case of least squares regression. We prove that, for the least squares estimator, the move from Fixed-X to Random-X prediction always results in an increase in both the bias and variance components of the prediction error. When the covariates are normally distributed and the linear model is unbiased, all terms in this decomposition are explicitly computable, which leads us to propose an extension of Mallows’ Cp (Mallows, 1973) that we call RCp. While RCp provides an unbiased estimate of Random-X prediction error for normal covariates, we also show using standard random matrix theory that it is asymptotically unbiased for certain classes of nonnormal covariates. When the noise variance is unknown, plugging in the usual unbiased estimate leads to an approach that we call R̂Cp, which turns out to be closely related to the existing methods Sp (Tukey, 1967; Hocking, 1976), and GCV (generalized crossvalidation, Craven and Wahba 1978; Golub et al. 1979). As for the excess bias, we propose an estimate based on the well-known “shortcut-formula” for ordinary leave-one-out cross-validation (OCV), resulting in a hybrid approach we call RCp. We give both theoretical arguments and numerical simulations to demonstrate that this approach is typically superior to OCV, though the difference is usually small. Lastly, we examine the excess bias and excess variance of other estimators, namely, ridge regression and some common estimators for nonparametric regression. The surprising result we get for ridge is that, in the heavily-regularized regime, the Random-X prediction variance is guaranteed to be smaller than the Fixed-X variance, which can even lead to smaller overall Random-X prediction error.